HDDS-10633. Support Content-MD5 header for checking object integrity when uploading object #9612
Conversation
|
Note:
|
|
Thanks @jojochuang for pointing this out. I agree that the final ETag may not always be an MD5 (especially for multipart or encrypted objects). However, this patch focuses on in-transit integrity checking via the During the upload, S3G calculates the MD5 of the incoming data stream to verify it against the client-provided |
# Conflicts: # hadoop-ozone/integration-test-s3/src/test/java/org/apache/hadoop/ozone/s3/awssdk/v2/AbstractS3SDKV2Tests.java # hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java
| String eTag = DatatypeConverter.printHexBinary(digest).toLowerCase(); | ||
| String clientContentMD5 = getHeaders().getHeaderString(S3Consts.CHECKSUM_HEADER); | ||
| if (clientContentMD5 != null) { | ||
| CheckedRunnable<IOException> checkContentMD5Hook = () -> { | ||
| S3Utils.validateContentMD5(clientContentMD5, eTag, key); | ||
| }; | ||
| ozoneOutputStream.getKeyOutputStream().setPreCommits(Collections.singletonList(checkContentMD5Hook)); | ||
| } | ||
| ozoneOutputStream.getMetadata().put(OzoneConsts.ETAG, eTag); |
There was a problem hiding this comment.
Nit: Since now MD5 hash is used for both ETag and content-md5 verification, let's rename variable eTag to md5Hash. This also applied to other places.
Let's also change getMessageDigestDistance and ETAG_PROVIDER to something like getMD5DigestInstance and MD5_PROVIDER.
| assertEquals("\"37b51d194a7513e45b56f6524f2d51f2\"", getObjectResponse.eTag()); | ||
| } | ||
|
|
||
| @Test |
Yes, ETag can technically be any hash that based on the object data so it doesn't need to be MD5. Moreover, object uploaded using multipart upload ETag is a hash of all the ETag of all the parts with the "-<num_parts>" prefix, see If we decide in the future to change ETag to not be md5 hash, then we need to setup a new MD5 |
3 participants
What changes were proposed in this pull request?
Add support for the
Content-MD5header to verify object integrity during object uploads.Please describe your PR in detail:
Content-MD5header using pre-commit hooks before the key is committed.Content-MD5Content-MD5KeyDataStreamOutputto simplify unit test setup.What is the link to the Apache JIRA?
https://issues.apache.org/jira/browse/HDDS-10633
How was this patch tested?
https://github.com/hevinhsu/ozone/actions/runs/20842334193